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The Impact of a National Evaluation Effort on the 
Utilization of Evaluation in Local Settings 



I. Understanding the Context for a National Evaluation Effort 
A. Background 

The National Network of Eisenhower Regional Consortia and Clearinghouse is composed 
of ten regional consortia and one national clearinghouse. Funded in 1992 by the U.S. 
Department of Education (ED) Office of Educational Research and Improvement 
(OERI), to help improve mathematics and science education throughout the nation, each 
of the ten Eisenhower Regional Consortia serves an ED regional educational laboratory 
region. There is a single Eisenhower National Clearinghouse (ENC), located at The Ohio 
State University. The Network sponsor strongly encouraged the consortia and 
clearinghouse to work together in close national collaboration from the beginning. This 
collaboration has included a substantial evaluation component; the consortia and 
clearinghouse decided early out that producing national evaluation results would be of 
value for both program management and survival. 

The consortia are specifically charged to: (1) provide technical assistance for the 
implementation of teaching methods and assessment tools for use by elementary and 
secondary students, teachers, and administrators; (2) disseminate exemplary mathematics 
and science education materials; and (3) build networks among mathematics and science 
resources within their regions and nationally. The clearinghouse collects, describes, and 
disseminates K - 12 mathematics and science materials on an extensive scale nationally. 

As a result of the Government Performance and Results Act of 1993 (GPRA) the 
legislation requires federally funded programs to submit performance reports and plans. 
Performance indicators set a standard for measure of success against a program’s stated 
goals and objectives. The ten regional consortia currently have eight performance 
indicators that address the overall objective to provide high-quality technical assistance, 
three performance indicators that address the overall objective to disseminate 
information, and one performance indicator that addresses the overall objective to build 
networks. 

The consortia and the clearinghouse, individually and as a collaborative network, conduct 
an intensive internal evaluation to document and assess their activities and impacts, and, 
as importantly, to provide management information for strategic decision-making. The 
internal evaluation effort has been supplemented by several third-party evaluations 
sponsored by ED. 
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B. Audience and Purpose 

This evaluation effort is addressed primarily to sponsors and partners of the consortia and 
clearinghouse network, as well as the staff of these organizations. It focuses in-depth on 
training/technical assistance, dissemination, and collaboration/networking activities and 
is organized around OERI’s Performance Indicators for the consortia and clearinghouse. 

It describes the activities and impact of the Network and also highlights areas for 
improvement of services. 

The major purpose of this evaluation effort is to assist the ten individual consortia, the 
clearinghouse, and the Network to reflect on their work in order to improve their 
practices and impact. Specifically, the evaluation is intended to describe the client 
service activities across all ten consortia and the clearinghouse, document the 
contribution to the improvement of mathematics and science education by assessing 
client perceptions of quality and impact, and stimulate recommendations for improving 
services and the evaluation itself. Also, consortia partners and relevant policymakers 
may find the evaluation useful in determining their desired role in connection with the 
consortia and clearinghouse to support the improvement of student performance in 
mathematics and science. 

B. Design, Data Collection Methods, and Products 

In 1993 the Network established a standing evaluation committee. The evaluation 
committee included one or more evaluators from each Network organization, as well as 
six of the directors. The committee meets during the summer to revise designs and data 
collection procedures and in the winter to conduct analyses and draft the annual report, 
which is edited and reviewed in subsequent months. Evaluators also serve on 
subcommittees to complete work between the meetings. The evaluators communicate 
between the national meetings by way of an electronic list. The list is used to send 
information and material, such as agendas and minutes of the meetings, data, analyses, 
and drafts of reports, as well as to discuss issues that arise around national evaluation 
work. 

The committee prepared a national evaluation plan including both descriptive and 
assessment-of-quality components. That plan was put into place in October 1994 by 
launching a pilot-test year of quantitative data collection using the Consortia and 
Clearinghouse Descriptive Data System (CCDDS). The data system was extensively 
revised based on the pilot test, and the refined system has been in place since October 
1995. The database describes each client-service activity conducted by the Eisenhower 
Network in terms of participant affiliation and role (e.g., teacher, administrator, 
policymaker), length of the activity, type of service provided, content focus, contact 
method, and involvement of collaborators. 

In 1997, a preliminary version of a national participant survey was developed and piloted 
in order to enact the assessment-of-quality component of the evaluation plan. The survey 
was designed to gather outcome and impact data from the recipients of products and 
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services, with particular interest in effects on professional practice and student 
performance. Revised versions of the survey were administered in 1998, 1999, and 2001. 
Approximately 1,000 surveys were completed in each administration. 

In 2000, an in-depth telephone interview procedure was developed and implemented in 
order to collect more qualitative information on impact. In all, 163 interviews were 
completed across the Network in the fall of 2000. The results were coded for access by 
theme and used as the primary source for this report. The principal impacts noted by 
respondents are discussed and illustrated extensively by quotations from clients. 

The Eisenhower Network published rudimentary descriptive reports in 1997 and 1998 
that depicted activities anecdotally and included limited data about the type and number 
of activities and participants, or clients, nationally. The Network also has published 
fairly extensive evaluation reports for 1996 - 97 (May 1998), 1998 (May 1999), 1999 
(June 2000), and 2000 (June 2000). These later evaluation reports included 
comprehensive, detailed descriptive information and participant opinion data about the 
usefulness, quality, and impact of Network services, products, and partnerships. The 
evaluation reports also described outcomes in terms of the OERI performance indicators, 
as required by funding regulations. 

C. Evolution of the National Evaluation and Present Questions 

The national evaluation of the Eisenhower Network has evolved greatly from 1993 to 
2002. In the early years, establishing common definitions and data collection procedures 
were major challenges. Selecting, refining, and implementing databases that could be 
used by 1 1 Network organizations for common analyses, as well as for individual 
purposes, has been a continuing issue and major consumer of resources. Generally, 
though, the national evaluation effort has resulted in a shared experience where problems 
are worked out successfully and the issues evolve to a higher level. 

The impact of the national effort on the individual organization evaluations is one of 
these higher-level issues, and is the topic of this session. In the early years, most of the 
evaluation resources went to individual organization data collection, analyses, and 
reports. Then, as the OERI Performance Indicators came along and grew in importance, 
the national effort demanded and received increased attention, to the point where the 
national design became preeminent. Since so many resources are now devoted to it, the 
question arises as to how this national effort is benefiting the organizations individually. 
What are the benefits? What are the barriers and facilitation to deriving benefits? How 
can the national effort be improved? 

E. Study Methods 

Several members of the national evaluation committee volunteered to study these 
questions and report on the findings and implications. The study was designed to include 
the ten consortia; the clearinghouse was not included because its performance indicators 
and evaluation reporting requirements are different. A symposium presentation at the 
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2002 AERA Annual Meeting was selected as one desired reporting medium for its value 
as a contribution to the field of evaluation. Two primary methods were used to prepare 
this symposium; they focused on staff interviews and document review. The interviews 
consisted of seven open-ended questions that examined: 

1. Familiarity with the work of the National Network of evaluators 

2. Alignment between the audience and purposes of the national evaluation and the local 
consortium evaluation 

3. Relationship between the national and local evaluation work 

4. Influence of findings from evaluation on policy and practices of the individual 
consortia 

5. Barriers and facilitators to using evaluation findings in the local organization 

6. High-priority needs in evaluation at the individual consortium level 

7. Ways of improving the national evaluation efforts 

The intent was to interview the director, evaluator or evaluators, and at least two program 
staff members from each consortium. The experience of the respondents as staff with the 
consortia ranged from less than one year to the initial funding of the Network in 1992. 

All interviews were conducted by telephone and lasted an average of 40 minutes. 
Verbatim responses were captured in writing as much as possible. Interviews were 
completed with the following consortium staff: 

• 10 directors (all consortia represented) 

• 1 1 evaluators (two from one consortium) 

• 1 5 program staff (one consortium was not represented) 

The documents reviewed included the 1998 - 99 annual report and five-year final report, 
which covered 1995 through 2000, from each consortium and the national Network. 
These documents were reviewed with the following interests: 

1 . Alignment of reporting periods with the national report 

2. Comparison of table of contents topics 

3. Number and content of appendices 

4. Comparison of evaluation methods 

5. Comparison of reporting evaluation findings, text, and graphic displays 

6. Comparison of reporting on indicators 

7. Comparison of conclusion sections: similarities and differences 

Evidence was sought concerning alignment of the evaluation methods, tools, and 
reporting between individual consortium evaluations and the national evaluation. Finally, 
the “lessons learned” sections and future evaluation plans in the national reports were 
examined with respect to implications on how to improve the quality and relevance of the 
national evaluation efforts. The document analysis was a secondary data source to the 
interviews. 

The results of this study will be presented and discussed by the panel members. 
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II. Definitions and Examples of Utilizing Evaluation in the National Network 

A. Levels of Awareness of National Evaluation Efforts , 

Each of the interviews began by assessing the respondent’s level of awareness of the 
work of the national evaluation network. This was done in order to contextualize their 
responses to subsequent questions. Respondents were asked the question, “How aware 
are you of the work of the national network of consortia evaluators?” Following this 
question, respondents were also asked whether they were aware of the national network’s 
three main data collection components — the relational activity-tracking database, the 
client survey, and the client interview. They were also asked whether they had read any 
of the national evaluation reports. 

As might be expected, organization directors and evaluators, on the whole, replied that 
they are very aware of the work of the national network of consortia evaluators. 

Underlying the directors’ awareness is the fact that six of the 11 consortia directors 
regularly participate in the national meetings of the evaluators. In addition, one director 
serves as the chair of the national evaluation committee and summarizes the joint actions/ 
and recommendations of the consortia evaluators to the directors on a periodic basis. 

As for the consortium evaluators, all of them, as mentioned in the previous presentation, 
are actively involved in the national evaluation committee, and the interviews confirmed 
that they tend to be very informed about the design and implementation of all the national 
evaluation components. As described in the previous presentation, national meetings of 
the network evaluators as well as ongoing conversations held via an electronic discussion 
group, or listserv, keep evaluators aware of and involved in the planning and 
implementation of national evaluation efforts. 

In addition, nearly all of the evaluators have been involved at one time or another in task- 
oriented subcommittee work that involves the detailed planning of design and 
methodology for one or more national evaluation components such as surveys, 
interviews, and an activity-tracking database. However, as the task of maintaining and 
periodically updating the activity-tracking database is, at some consortia, delegated to 
other consortium staff members, some of the evaluators noted that they are less aware of 
issues related to the design and use of the database. Another possible factor that would 
explain this is the fact that the structure of activity-tracking database was redesigned 
recently, and thus evaluators and staff have had only approximately four months to 
familiarize themselves with its new features. 

With regards to program staff, all of them said that they are aware or somewhat aware of 
the work of the national network of consortia evaluators. When they were asked more 
specifically about their familiarity with the various national evaluation components, they 
tended to indicate varying degrees of familiarity with the activity-tracking database, the 
national client survey, and the client interview. Very few of the program staff had read 
any of the national evaluation reports. 
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B. Influence of the National Evaluation on Local Evaluation Activities 

How has the work of the national evaluation influenced local evaluation efforts? One way 
of gauging this relationship is to determine the extent to which evaluators perceive their 
national evaluation work as overlapping with their local evaluation work. When 
evaluators were asked what proportion of their time is allocated to national, or cross- 
consortium, work as opposed to local consortium evaluation work, the responses varied 
greatly. A few evaluators remarked that they devote approximately half of their time to 
national evaluation work and half to local evaluation work, some of them said that more 
of their time is focused on the national evaluation work, and others said that less of their 
time involves national evaluation work. 

Perhaps not surprisingly, several evaluators noted that their national evaluation work 
overlaps to some extent with their local evaluation work. The explanation for this is that 
when each evaluator collects data to feed into the national evaluation work - be it client 
survey data, client interview data, or activity data for the database - this data is derived 
locally, and the instruments used to collect the local data that feed the national evaluation 
are instruments developed by the national network, or adaptations thereof. Furthermore, 
as part of the national evaluation effort, evaluators are occasionally asked to perform 
some degree of analysis on those locally derived data that feed into the national dataset. 
This may contribute to a perception of overlap. 

However, a perception of overlap between the national and local evaluation work is not 
sufficient, in and of itself, to assess the extent to which the national evaluation work has 
influenced local evaluation efforts. To gain a clearer picture of this relationship, interview 
respondents were asked the question, “Can you give examples of how the work of the 
national network of evaluators has influenced the evaluation of the local consortium 
or of your organization?” 

In general, the interviews with consortia directors, evaluators, and program staff revealed 
that, across the network, the national evaluation work tends to influence local evaluation 
efforts in some very specific ways: 

• For many, the national evaluation work is viewed as helping to provide a more 
formal structure to local evaluation activities, and in some cases is even described as 
being the motivating force behind the design and implementation of local evaluation 
efforts. As the most recent example of this, several respondents mentioned that the 
national evaluation has influenced the way in which individual consortia measure and 
report on the impact of their intensive work at school sites. As another example, the 
national network report is used, to varying degrees, as a model to help structure the 
format and content of the local progress reports. 

• Possibly the most important factor underlying these influences, pointed out by many 
respondents, is the Network’s set of performance indicators, which were required 
by the Government Performance and Results Act (also known as the “GPRA 
indicators”). As mentioned in the previous presentation, these indicators were 



established and adopted nationally by the Network to measure and report on its 
progress toward 12 performance benchmarks, and a subset of these indicators was, in 
turn, adopted by each consortium. As a type of accountability system to Congress and 
the U.S. Department of Education, the indicators have ultimately served as an 
evaluation framework guiding the national as well as the local evaluation efforts. The 
adoption of this system has had the effect of motivating the consortia, individually 
and as a national network, to collect data that can be compared to the indicators in 
order to measure progress. Over the years since adopting the indicators, the national 
evaluation effort has striven to measure the Network’s progress against these 
indicators by developing, refining, and implementing the necessary data collection 
tools and processes described previously — including the activity-tracking database 
system, the client survey, and the client interview, in addition to procedures for 
evaluating intensive-site work. And in turn, as was revealed through interviews with 
consortium directors, evaluators, and program staff, these same tools and processes 
tend to be used at the local level to measure and report on the progress of individual 
consortia. For example: 

o Many local evaluations have begun implementing a classroom observation 
protocol that was shared as part of the national evaluation effort, as a tool to 
collect classroom impact data from sites where intensive work is carried out. 

o The client survey instrument developed through the national evaluation work has 
provided, in the words of one director, “one of the best longitudinal looks at what 
we’ve been doing, that we’ve ever had.. .it really changes the way we think about 
our data [at the local level].” 

o The newly redesigned activity-tracking database developed through the national 
evaluation is also used on a local level to document work. The use of this database 
has motivated a more in-depth level of data collection on clients, events, and 
product dissemination; promoted a common understanding of how terms such as 
“activity,” “intensive event,” and “focus area” should be defined; highlighted the 
need for local as well as national consistency in data collection efforts; and, 
finally, allowed staff to document their complex, multilayered work like never 
before. 

• The national evaluation work is seen as supporting the goals and activities of local 
evaluation to a significant extent, at least with regards to measuring and reporting 
progress. For example, one program staff member noted that the national and local 
evaluation are “mutually supportive. Our own consortium has learned from the 
national group and vice versa.” However, because the national evaluation work is 
seen as being more summative in nature, local-level evaluation efforts are typically 
complemented and buttressed by other measures, such as workshop evaluations, 
targeted surveys, observations, and informal feedback from clients, which are 
considered to yield data that is more formative. 
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• To many directors and evaluators, the demands of the national evaluation work have 
grown over the years, and for some consortia this has prompted a greater investment 
of resources into evaluation, in terms of staff time. 

• It appears that thus far the investment of time and resources into the national 
evaluation effort has, on the whole, benefited the local evaluations. As described 
earlier, some evaluators commented that the national evaluation work facilitates their 
local work by giving them the tools and methodologies they can use to collect local 
data. Some evaluators mentioned that they value the collegial conversations and the 
opportunities to exchange ideas and strategies that are made available through this 
evaluation network. Other respondents remarked that the national evaluation work 
has contributed to the rigorousness and even enhanced the quality of local 
evaluation efforts. For example, one director commented, 

I think the quality of the group effort has really influenced the quality of our 
individual effort. 

• The national evaluation has also, for many, resulted in a greater awareness of the 
importance of data and the need to look closely, in a more formalized way, at the 
impact of program work and how this aligns with program objectives. According to 
one respondent, the national evaluation “helped focus staff on the importance of data 
and what we might leam from the data,” while another program staff member noted, 

We 've become more rigorous in our own evaluation. [We are becoming] clearer 
about what our objectives are as projects, and monitoring those and paying attention 
to the data more. 

• The indicator framework as well as the tools and processes of the national consortium 
network evaluation have begun to serve as a model for other labs and regional 
initiatives, as well as national organizations that function through regional centers. 

• The review of the consortia annual reports revealed that all consortia use the CCDDS, 
the survey, and interviews as part of their evaluation report. Each consortium 
reported its standing on the Performance Indicators. About a third of the consortia 
used the identical formatting and graphic display as that used by the national report. 
For example, in the five-year report, many consortia used the vignettes gleaned from 
their interviews that had contributed to the national report. 

C. Influence of Evaluation Findings on Local Policy and Practice 

How have the evaluation findings, whether at the local or national level, influenced local 
policy and practice? Interviewees were asked whether they could give examples 
illustrating how either the national or the local evaluation findings have influenced 
local policies or practices in any way. For all of the consortia, the national evaluation 
findings as well as the local findings generated through implementation of the nationally 
developed data collection tools (i.e., the activity-tracking database, the client survey, and 
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the client interviews) seem to be most useful in verifying whether the consortia, either 
individually or collectively as a network, are meeting their benchmarks of success — 
i.e., the GPRA performance indicators — and they serve to validate and substantiate that 
local endeavors are moving in the right direction. Along these lines, many respondents 
commented that they feel the primary purpose of the findings culled from the national 
evaluation work is to show accountability. With regards to whether or not the national 
evaluation findings have been useful in a formative way, interviewees shared mixed 
perceptions. 

The national evaluation findings do seem to inform some strategic decision-making, but 
to varying degrees. Some respondents commented that the national evaluation work has 
helped consortia to see and reflect on the “big picture” and “overall impact” of their 
work. As one director remarked, 

For us... it 's some of the first data that we 've had which has allowed us to look at our 
work in a complete manner... this is the first summative data that we 've been able to 
use. 

In addition, this director noted, the mid-point review and subsequent report resulted in the 
consortia becoming more thoughtful and strategic about their services. Other directors 
echoed this theme; one said that the client survey responses were discussed in meetings 
and raised awareness of the “need to rethink the way we’re doing our work.” Another 
director cited the value of the national evaluation reports, which helped him to 
“understand what others have done in the past, to gain insight from their challenges and 
successes, such as the need for student achievement data to demonstrate our performance, 
and the difficulties in collecting that data. That helped me to think through those issues 
and be aware of them and gain insights and draw on lessons learned from colleagues.” 

For many respondents, the local use of the indicator system, the activity-tracking 
database, the client survey, and the client interview have had the effect of making staff 
more aware of the type of data that is being collected, and what the expectations are. One 
program staff person mentioned that the national evaluation findings and local findings 
drawn from the national data collection measures make staff aware of program strengths 
and weaknesses and thus “helps us be smarter about our work.” 

Other staff cited more specific and direct effects on local policy and practice. In 
particular: 

• Several respondents mentioned how their use of the database and adoption of the 
indicator system have made them more aware of the importance of intensive 
activities and how this has impacted the nature of their training and technical 
assistance. This awareness has tended to prompt a shift away from the so-called “one- 
shot” workshops and a greater emphasis on providing institutes that last two or more 
days. As one evaluator described it, 
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National evaluation reporting on the indicators has influenced our policies: seeing 
low performance on the intensity indicator prompted a stricter requirement with 
projects that we support financially. We started requiring [more intensive work from] 
state teams, that any effort we support had to be more than 12 hours. 

• Several staff members mentioned that the indicators have helped foster a greater 
emphasis on serving schools with high proportions of at-risk students. This has 
stimulated discussions about the types of schools to which local services need to be 
targeted, and in several places has already impacted, or will likely impact, the 
selection of school sites where training and technical assistance is delivered. 

• In a few places, the national findings have brought about a shift in the nature of 
training/technical assistance being provided. As a result, one consortium is 
becoming more involved in school-site-based work and moving away from the 
training-of-trainer model, whereas another consortium, in contrast, has begun 
focusing “more energy on building support for regional resource providers, and less 
energy [on] direct service to schools.” 

• A few respondents described how the national evaluation work has made them more 
aware of the importance of partnerships and collaborations and encouraged them 
to seek out ways to develop partnerships with key stakeholders. 

• One respondent described how the client survey results brought greater attention to 
the need to strengthen the centers of product dissemination and subsequently 
motivated consortium staff to provide more assistance to the staff at access centers. 

• Another respondent mentioned that she examines client survey findings on a state-by- 
state basis and uses those findings to generate discussions and guide planning 
between state team members. 

According to a handful of respondents, however, the national evaluation work has not 
influenced local policy or practice, and some went on to attribute this to the emphasis on 
accountability that has been instilled through the required adoption of the GPRA 
indicator system. As one respondent put it, “The primary purpose for gathering data is to 
answer the department indicators,” while another respondent noted that “the value is not 
positive for impacting programs” because the national evaluation, by necessity, is 
mandated to produce quantitative data that can be compared to the indicators. In 
alignment with these comments, some program staff indicated that the evaluation 
findings that are most useful in guiding the planning and implementation of their work, 
on a day-to-day basis, is data that is collected locally at the initiative of the consortium, 
as a supplement to the data that is collected for indicator-related accountability purposes. 
As mentioned earlier, these data typically take the form of workshop evaluations, targeted 
surveys and interviews, observations, and informal feedback received on an ongoing 
basis from clients. Several respondents cited ways in which these types of data sources 
have contributed to program planning. For example, program staff members described 
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how data from workshop evaluations have brought about modifications in the way 
professional development is delivered, to better address the needs of participants. 

D. Conclusion 

In sum, the overall trends of responses to these two interview questions suggest that the 
national evaluation work has been valuable in a variety of ways. It has guided the 
methodology and reporting of local evaluation efforts by providing tools and processes 
that can be used or adapted locally to measure program success. The national evaluation 
work has also served to inform and influence local policy and practice, although for 
some, its utility in this regard is perceived to be somewhat limited due to its emphasis on 
accountability. 

Perhaps the lesson to be learned is that the most powerful evaluation is one that combines 
(1) nationally developed tools and processes to measure accountability, with (2) locally 
derived evaluation strategies that are designed to gather formative data for the day-to-day 
decision-making needs of program planners. An example of how these two approaches 
can effectively be joined together was provided by a program staff member in the 
following comment: 

At the end of the year, we discussed the intensive site with the evaluator. We had 
qualitative and quantitative data. The evaluator had quantitative data (from the national 
activity-tracking database) and people working on site had quality data, and we had a 
lengthy discussion on how to make improvements and whether or not to drop a site, and 
we used evaluation to help us [make this decision] . 

This example of decision-making that is based on multiple data sources and draws from 
the national evaluation work might be considered to represent the ideal. However, the 
degree to which this ideal is realized on a local level may not be as extensive, in reality. 
Consequently, the next part of this presentation is devoted to examining the various 
barriers and facilitators to making the best use of evaluation findings — in other words, the 
barriers and facilitators that may affect the extent to which consortia can achieve this 
ideal. 

TTT Perceptions of Facilitators and Barriers to Utilizing Evaluation in Local Settings 
A. Overview 

One aspect of our examination of the Network’s utilization of evaluation findings focused 
on determining elements of the national network system, as well as local consortium 
issues, that serve to facilitate or to act as barriers to the utilization of evaluation findings. 
During our interviews with directors, evaluators, and program staff, we asked the 
following questions to investigate these issues: 

• To what extent are the purposes and audiences of the national evaluation aligned with 
the evaluation of your consortium? 
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• From your perspective what are the barriers to using evaluation findings in your 
consortium, and what are the facilitators? 

This presentation describes the information that was gained from these interview 
questions. There are two purposes in our investigation of facilitators and barriers. In 
essence, the first purpose is to gain information to improve the “fit” between national and 
local evaluation needs. In this sense, we needed information that could serve as a “meta- 
evaluation” and would provided us with information to help us make critical decision to 
improve the work of the Network evaluators to facilitate the utilization of evaluation 
findings. The second purpose is that, through our meta-evaluation, we hoped to gain 
information to help other national organizations that would like to embark on a similar 
cross-organizational evaluation network. 

The major categories that emerged from the interview analysis regarding facilitators of 
utilization of evaluation findings were: 

• alignment of audience, purpose and focus, and data collection 

• the Network as a resource and a learning community 

• program directors’ support 

On the other hand, the major categories that emerged regarding barriers to the local 
consortium’s utilization of Network evaluation findings were: 

• issues of time: time commitment, timelines, trade-offs, timeliness, and time for 
reflection 

• useful common denominators 

The following sections describe the findings and issues in depth and help to provide 
insight into the complexity of aligning national and local evaluation efforts. 

B. Facilitators of Network Evaluation Utilization 

• Alignment of audience, purpose and focus, and data collection 

Alignment of audience . One consistent theme that was mentioned as a facilitator for 
the utilization of network and local evaluation findings was the fact that Network and 
local reports have a similar audience. Although some of the respondents mentioned 
that their local consortium must also consider specific local stakeholders when they 
produce their reports, all of them talked about the need to send reports to the OER1 — 
the same audience for the Network report. One evaluator explained. 

The Network evaluation audience is the Department of Education. The hidden 
purpose is to please them. The prime purpose is to show that we are leveraging 
resources and providing technical assistance and that we are adding value to math 
and science teaching and learning. That is on the national level. My audience here is 
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parallel. We need to produce reports that go to the government. The national and 
local evaluations share our audience and our clients. 



Alignment of purpose and focus . In addition to audience, the Network and local 
consortium reports share similar purposes and foci, “the purpose being to assess . . . 
progress toward goals.” Most importantly is the need for both the Network and local 
consortium to provide information about their progress toward meeting OERTs 
indicators. One director explained, 

The indicators became the glue for us ... We have to answer those questions as a 
[national] program as well as an individual consortium. 

With a focus on addressing the progress of meeting the indicators at both levels, the 
director pointed out, 

We would be working smart if [our plans] play hand-in-hand. 

Alignment of data collection . By focusing on collecting data that provides indicator 
information, data collection methods were designed by the Network to collect 
systematically and uniformly local data that could be aggregated across the consortia 
to inform the government about the Network’s progress toward meeting the 
indicators. Over the years these methods have become more and more refined. Both 
the directors and evaluators spoke highly of the descriptive data system that has been 
developed, the surveys used to collect client perceptions about the quality and utility 
of local consortium products and services, and the recent interviews to gain more in- 
depth understanding of client perceptions. The local data is fed into the Network 
evaluation, but it is also used locally to inform each consortium about individual 
progress toward meeting the indicators. As one director explained, 

The two evaluations are closely aligned and have become more aligned in the last few 
years. That alignment has really been pushed by the fiscal reality that we don 't have 
a lot of evaluation dollars to go around. So, basically what we ve done is taken the 
national evaluation as being the core of the evaluation that we re going to conduct 
for our individual consortium as well... We really use the national plan as our core. 
So... we use our consortium portion of the CCDDS to print out all of the descriptive 
tables and information that we use in our individual consortium report. We also use 
the survey and interview responses from the national design; we analyze them, 
separately just for our individual consortium reports. 

Because there is an alignment between the Network and local evaluation purpose and 
focus, data collection, and audience, many of the local reports mirror the Network 
reports, which “inform... decisions about how to write the report,” in organizational 
structure, the descriptive tables, or other features. This saves time and energy for 
those who create the local report; they do not need to “reinvent the wheel.” 
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The alignment of the Network and local consortium’s purpose and audience 
facilitates evaluation data collection, interpretation, and ultimately, the use of the 
findings. 

• The Network as a resource and a learning community 

The Network as a resource . One major benefit of the Network is that it serves as a 
resource for all members who are part of it. One director stressed, 

The Network serves as a great resource for all our work, in that we would be out here 
working in isolation if we didn 't come together as a Network to see how our work can 
enhance each other's work. 

Together, the Network members discuss and refine the definition of terms (e.g., the 
definition of “activity,” “event,” and “at-risk”), the data collection methods and 
instruments, and the preparation of national reports. Although the focus is usually on 
addressing the indicators at a national level, because the audience, purpose, and data 
collection are aligned with local efforts, the refinement of evaluation terms, methods, 
and reports at the national level means refinement at the local level. Indeed, the 
Network evaluation work and methods have evolved over time to be more useful and 
more aligned with local needs and efforts. One evaluator elaborated: 

The fact that we 're all trying to focus in the same direction and talk about some of the 
issues does facilitate better evaluations locally. At the national level, we discuss 
“What is an activity? ” and “How are we going to measure our work? " This 
conversation helps feed back into the local evaluation to make it better. 

Leadership is another resource. In order to produce the Network evaluation plans, 
tools, and reports, evaluators take leadership roles. Although some consortium 
directors participate in the meetings, it is the evaluators who sort through the 
technical issues, propose solutions, and develop the tools. Through leadership and 
collaboration the Network has evolved to be a great resource. One director 
commented that the growth in leadership was tied to the pride that evaluators take in 
their work, with the overall result that building relationships enhances the quality of 
the evaluation procedures and products: 

I think one of the things you see is the growth of the group, the leadership that they 
take, and the pride they take in their work. That, to me, is always important. 

The Network as a learning community . Perhaps one of the most important aspects of 
the Network is the ability of members to leam from each other. This aspect, one 
director explained, has been an evolutionary process that has made the Network 
evaluation team become a learning community: 




15 



17 



You learn from one another, and that 's been a facilitator for the work, in that you 
come together as a group and learn from each other. It ’s a learning community for 
me as well as [a group producing] real work. 

The learning-community aspect of the Network facilitates evaluation quality at 
s national the level, which in turn facilitates evaluation quality at the local level. 

• Program directors’ support 

Not surprising was the fact that evaluators and program staff stressed the importance 
of the directors’ role in facilitating the utilization of evaluation findings. The 
interviews suggested that “having a director who understands and supports evaluation 
and evaluation findings” was one of the most important facilitators for utilizing 
evaluation findings at the local consortium. Directors are “very influential” when they 
model “consistent support... for the importance of evaluation,” because “commitment 
on the director’s part. . .gives the general message of what’s important in the 
organization.” 

Communication seemed to be the key to utilization of evaluation findings. The most 
important of directors’ actions that were seen as facilitating the use of evaluation 
findings included having regular staff meetings (e.g., biweekly) attended by 
evaluators, where a portion of time was set aside to discuss evaluation data collection 
needs, program evaluation questions, and evaluation findings. One evaluator gave an 
elaborate description of her involvement with the program staff: 

lam regularly at staff meetings ...discussing evaluation results and talking about 
evaluation data collection, what I'm going to be doing in the field, what I’m going to 
be asking of their clients, and asking for their feedback on who and where to sample. 
And in the case of the intensive sites — [going] over the survey items that we use to 
make sure, from their perspective, that the questions get at what ’s going on at their 
school. ..So there’s a close level of communication between me and the staff, 
supported by the director. The director really facilitates the use of evaluation 
findings. 

Another important aspect that was coordinated by the director was the level of 
evaluator involvement with project activities. From program staff s point of view, 
being able to utilize evaluation findings begins with “having an evaluator who 
understands both the nature of the work and the structure of the evaluation so the 
links are really clear and strong and can be communicated to program staff.” The 
evaluators’ understanding of programmatic work is facilitated by directors who allow 
the evaluators to be “engaged in activities. . .so [evaluators] have first-hand experience 
[that would allow them to] both collect and interpret data better as well as to make 
decisions about what other data can be collected.” When directors orchestrate their 
program staff and evaluators to work together collaboratively, the evaluation findings 
are better utilized. 
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C. Barriers to Network Evaluation Utilization 



• Issues of time: time commitment, timelines, trade-offs, timeliness, and time for 
reflection 

The most frequently mentioned barriers to the utilization of evaluation findings 
revolve around issues associated with time. These barriers include: the large amount 
of time needed to conduct the Network evaluation activities, the misalignment of 
national and local timelines, the scarcity of evaluation resources necessitating trade- 
offs among evaluation activities, the difficulty of providing timely feedback to 
directors and program staff to help with decision-making, and time for the directors 
and program staff to reflect on the findings and initiate programmatic changes. 

Time commitment . The national Network evaluation requires a substantial 
commitment of time to conduct. Fortunately, the Network data collection needs 
overlap with the local data collection needs. The more that the national and local 
evaluation needs overlap, the more efficient are evaluation efforts. However, beyond 
the data collection, the Network evaluation activities still require a fairly large 
amount of time. In addition to the several-day national evaluation meetings held twice 
each year, each requiring a considerable amount of time and effort to prepare, a great 
deal of time is expended with subcommittee communications and duties. Some 
subcommittees require more involvement than others, and that may create an 
imbalance in commitments among member organizations. Many directors and 
evaluators expressed hope that, as the Network continued to evolve, the 
subcommittee time commitments would decrease. This was particularly true of the 
subcommittee that was working to overhaul the descriptive data system. Still, the 
amount of time required to participate in the Network subcommittees took time away 
from local evaluation efforts. 

Timelines . One time-related barrier to the utilization of findings is that the national 
reporting timeline is not aligned with the local consortium reporting timeline. For 
example, the Network survey distributed during December investigates the quality 
and utility of work done during the previous funding year (September through 
October). The Network aggregates the data during the following January, and usually 
completes the national report in the spring, approximately six months after the close 
of the funding year studied. 

However, the local consortium reports are due in June and are based on evaluation 
data that is collected for the preceding May through April, in essence reporting on the 
current school year. The national survey data, however, is focused on the previous 
school year. At best, the local survey findings might help to make middle-of-the-year 
changes (e.g., in January) or to confirm that the activities done the previous year were 
on track. Thus, the findings may not be available when they could best be used to 
suggest programmatic changes (in the summer). One evaluator explained, 
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The timeframe of the reporting periods between the nation and local are not the 
same. [Therefore] the local reports get sliced us... and what we report to our contract 
officer are these odd chunks. Sometimes it 's a 5-month period; sometimes it ’s an 8- 
month period. [But] our national reports are always a full fiscal year. 

Trade-offs . The chances that evaluation findings can help in decision-making are 
increased as more comprehensive evaluation data are collected. But resources for 
evaluation are limited. One evaluator commented, 

[One] barrier goes back to the time-cost issue. Of course, the more money you can 
put into evaluation, the more time you can buy for evaluation [activities] , and the 
more information you can give back to the program staff in a formative sense to make 
[the project] better. Part of that whole equation becomes getting information back to 
them quickly, and all of that times time, and time is money. And [lack of evaluation 
funding is] a barrier for everyone. 

Program staff, especially those who use and value evaluation findings, would like to 
have more information to assist their work. When asked about the barriers to using 
evaluation findings, one program staff member explained, 

Perhaps the only concern is that we don t have enough [findings]. It just seems 
impossible to assess or create the research questions that go with everything that we 
do... Primarily the problem is sometimes we don ’t have enough information, we don ’t 
have enough data because of limited resources for evaluation... We have to prioritize 
the evaluation needs to address those [questions] that are most immediate, because 
there aren ’t enough resources to evaluate everything. Other than lack of resources, 
there isn 't anything keeping me or other staff from making the best use of the 
evaluation data. 

Timeliness ofjindings . Related to all of the time-related barriers is another barrier to 
the utilization of evaluation findings: the timeliness with which evaluators can 
provide information to directors and program staff. From the perspective of the 
directors and program staff, if the evaluation findings are not delivered in a timely 
manner, they are almost useless — at least within a formative timeframe. Most of the 
evaluators were highly aware of the need for timely reporting. However, many of 
them found another time barrier that hindered their efforts: so much data and so little 
manpower to prep and analyze data. For example, one evaluator said, 

We ’re using the Horizon teacher questionnaires with all of our intensive sites. They 
provide good information because they are so detailed. Yet, the data entry and 
analysis is more time-consuming because they are so detailed. Then, to get the 
findings into the hands of the people that are doing the work — there ’s not a quick 
enough turnaround. 

Time for inflection . Getting timely evaluation findings to directors and program staff 
is very important for sound decision-making; however, as evaluation data collection 
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increases, it becomes more difficult — in terms of hours needed — to turn the data into 
something meaningful and useful. When evaluators do give evaluation findings to 
directors and program staff, an additional time-related barrier emerges: time for 
reflection. As one program staff member stated, 

In a broad sense, time is a barrier - taking the time to understand evaluation 
findings, and not just recognize that they 've been done and then throw them in a pile 
somewhere, but to actually digest them and draw meaning out of them. I think we are 
making an effort to do that, but the time to do it is always a difficulty. That’s one issue 
that comes up repeatedly in our discussions: How do we make more effective time to 
share ideas together, to help each other do a better job with our clients at our 
respective sites? And part of that is, of course, using evaluation. 

• Useful common denominators 

Another barrier to the use of evaluation findings — especially findings at the Network 
level — is developing common denominators within the work of all the consortia. 
Providing information about the Network’s progress toward meeting the indicators 
involves a high-level, rather abstract, set of variable definitions. Therefore, findings 
may be applicable only at a high level of decision-making, such as determining the 
need to disseminate a greater number of publications. Providing information that is 
more useful to local program staff may require more in-depth, localized information. 
One staff member explained, 

We really do not have any [completely uniform] truly cross-consortia projects. If you 
think about it, the only thing that the ten consortia have in common is the fact that 
they are local consortia serving different regions. So the barrier is really that the 
national evaluation is about different projects ...It's hard to find the common 
denominator. 

Evaluators and directors talked about the fact that the Network evaluation findings — 
aggregated and disaggregated — did not always tie directly into programmatic work, 
thereby making the utilization of the Network data collection and analysis more 
difficult. One evaluator explained, 

For the findings to be useful locally, I think we would have to do our national 
evaluation more closely aligned to what staff are doing out in the field. I would say 
that the national [evaluation] is more global, more directed at collecting information 
on the indicators — which are useful to an extent, but not to the depth or specificity to 
what this consortium does that would be helpful in a formative way to the program 
staff. 

While the indicators provide cross-consortia evaluation guidance, the data collected 
to address indicator progress is global and summative in nature. For example, 
indicator-related data describe the number, and types, of clients served and clients’ 
general perception of the quality and utility of consortia work. But that type of data is 
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somewhat removed from the program staff working in the field. To make refinements 
in their work, they need more formative evaluation information that they can access 
quickly to make immediate decisions. 

D. Conclusions 

The interviews provided insight into the barriers and facilitators of utilizing Network 
evaluation findings at the local level. The predominate barriers identified by the 
respondents revolved around issues of time and lack of evaluation funding. Balancing 
Network and local evaluation activities requires an analysis of time and cost trade-offs. 
Getting timely formative evaluation findings into the hands of program staff is important, 
but it is often hindered by the workload involved in preparing and analyzing the data. 
Also, finding the common denominators among consortia work and making the national 
evaluation effort more closely aligned with local formative evaluation needs is also a 
challenge. Because the Network has evolved over time into a learning organization, these 
barriers and challenges may be overcome through the resources and collaboration the 
Network evaluation team provides. 

IV. Improving the Validity and Value of Evaluation between National and Local Components 

This presentation focuses on what we have learned from our study of the utilization of 
evaluation in the national Network to improve the quality and relevance of evaluation work. 
This presentation examines how the evaluators’ roles and responsibilities in the local 
organizations are changing, how the national evaluation activities influence the value and 
relevance of local evaluation, and if the impact is multi-directional. 

One of our hypotheses is that, for the national evaluation work to be relevant to the local 
consortia, we have to examine the high-priority needs of the local consortia for evaluation 
services in comparison with the national evaluation priorities, which are to report on the 
performance indicators that are aligned to the three major objectives across all consortia. 

A. High-Priority Needs with Regard to Evaluation of the Local Consortia 

We found that the responses, regardless of the role of the respondent, clustered into three 
broad themes. Each is discussed below. 

The need to improve the relationship between evaluation and program work . Many 
respondents spoke about the need to improve communication about the evaluation 
requirements and findings. They emphasized the need to: 

• get the findings back to staff in a timely manner so they can use them 

• make evaluation findings more meaningful to program staff in terms of timeliness and 
understandability 

• educate program staff on the importance of evaluation 

• keep evaluation in the forefront of everyone’s thinking 
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Program staff and directors noted the importance of being sensitive to staff relationships 
with clients. They asserted that: 

• there are trade-offs between doing evaluation and providing service to clients 

• evaluation needs to benefit the clients-schools, teachers, and students 

• it is important to find the balance between being a service-providing organization and 
doing evaluation, so evaluation isn’t an impediment or burden 

Program staff and evaluators reported that being more familiar with the consortia’s work 
would enhance the ability to design more relevant evaluations. They suggested: 

• more on-site work by evaluators in their consortium’s activities 

• more knowledgeable evaluators about their consortia’s work 

• that evaluators get a first-hand impression, to become totally familiar with all the 
consortium work 

The need to balance formative and summative evaluation tasks . One of the biggest 
challenges for evaluation is to provide feedback on how program staff deliver the 
services and how clients perceive the quality of the consortia’s services and products, as 
well as gather data on the outcomes of the work. These methods require different tools 
and data demands from staff and/or clients. Respondents saw the value in both, but the 
local evaluation clearly emphasized the need for formative evaluation. They suggested: 

• more formative evaluation to determine interim findings on benchmarks, such as a 
mid-year review, for a chance to change or modify 

• striking a balance between the summative evaluation efforts at the national level and 
the formative evaluation efforts at the local level 

• meeting the formative evaluation need at the local consortium, which requires 
evaluation designs for each of several projects 

Conversely, all respondents noted the need to find out if what consortia do makes a 
difference or has an impact on different clients in terms of the work with aligning 
curriculum, instruction, and assessment, providing professional development, and 
dissemination. They suggested: 

• making the connection between curriculum, instruction, and assessment; helping 
clients align curriculum, instruction, and assessment and evaluating how well we do 
that; determining whether we are really meeting the needs of people working with 
high-risk students; finding ways to measure whether we’re making a difference for 
teachers and students; discovering how kids benefit from school change efforts 

• determining how many people we are really impacting with multiple hours of 
professional development 

• looking at the impact of our work by creating a flow chart of the process and plan 
"with evaluation 

• focusing on impact on student achievement in intensive site (middle school math) 
work and the methods to show this impact 
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• revealing the impact of dissemination through print products, electronically; 
determining teasing out impact on dissemination materials from a variety of sources; 
determining what information people are getting and what we intend that information 
to do; discovering the impact on the effectiveness of dissemination activities 

• separating out impact on different clients, such as school leaders, state department 
leaders, or teachers 

The need to improve the local evaluation processes. Respondents noted specific 
evaluation needs that related to either making better use of existing tools and methods or 
creating better tools and methods. They claimed they needed: 

• time to really analyze qualitative data 

• use of the new CCDDS database to its full capacity as a relational database 

• more accuracy about client data, such as tracing high-needs information 

• to create a logical, goals-based evaluation system that connects overarching goals 
with programs and allocates evaluation to those programs, while integrating indicator 
information 

• to get a better sense of the work state-by-state, looking at the professional 
development infrastructure and what the needs are 

• to enhance tracking of dissemination electronically and from Access Centers 

• valid and reliable measures of student achievement to develop the tools, like student 
interviews and student voices, and processes for evaluation work at the school level 
that are “quick and dirty” 

• more resources for evaluation work 

B. Improving the Quality and Relevance of the National Evaluation Work for the Local 
Consortia 

We wanted to know from the respondents’ perspectives and experiences what would help 
close the gap between the local and national evaluation work to increase the utilization of 
evaluation. 

To assure commitment to collaboration . The “Future Evaluation Activities” sections in 
the last two national evaluation reports listed the responsibilities of the national network 
of evaluators as follows: to meet twice a year for design and analysis; to implement the 
CCDDS, surveys, and/or interviews; to generate a six-month formative report based on 
the CCDDS; and to conduct an annual comprehensive evaluation. 

One of the prominent themes from all the respondents was assuring the high quality of 
the national evaluation work, which is directly influenced by the commitment from 
directors and evaluators to do high-quality work. This means active support of the 
national evaluation by allocating the time and money at the local consortium level to do 
the work beyond just attending the meetings. Conversely, the lack of this commitment 
could result in reduced validity and reliability of the national work. Comments included 
the need for: 
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• open communication between directors and evaluators 

• understanding and support for evaluators from the directors (uneven buy-in across the 
national network) 

• directors’ commitment to collaboration, to increase validity of national findings 

• commitment to the national effort, as a lack of this lessens integrity of the work and 
lowers the morale of the group, which impacts on the validity 

• more efficient online discussion with higher expectations on connecting between 
meetings 

• greater commitment from all the directors that national activities are important; a 
realization that evaluators need support for the work to increase the consistency of the 
work 

• collecting the same data in the same way, since it gets aggregated at the national level 

• ensuring the consistency and quality of the national work and that there are common 
understandings and commitments to doing the national evaluation in a certain way 

® more accountability and agreement among the consortia for what data absolutely need 
to be collected 

• more cooperation and consistency and understanding among the evaluators and 
directors at the national level, being more egalitarian and taking differing opinions 
seriously 

• more work in smaller groups, as the larger groups tend to get sidetracked, while 
smaller groups can tap into expertise better 

Respondents described several enhancements that could positively affect the quality of 
the design, evaluation tools, data collection procedures, data analyses, and reporting 
results. One of these was a recognition that we need to build the professional capacity of 
the evaluators who have different levels of experience and skills. 

More professional development for evaluators. . Evaluators are not all at the same skill 
level in all aspects of evaluation. More professional development around the nature of 
evaluation work for both staff and evaluators would be helpful as the design and 
methodology get more complex. Respondents advocated: 

• providing means of developing more consistent skills among the evaluators 

• increased awareness by evaluators of the differences across the various consortia may 
help them design evaluation 

• concern about: the growth in the number of collaborators and maximizing the 
potential; the tension of an egalitarian structure when making sure all voices get 
heard, which takes up time; getting the work done, and making the hard decisions 

To balance summative and formative evaluations . Similar to the priority needs at the 
local consortia, many respondents talked about how the national evaluation was more 
summative (for the purpose of addressing the indicators, accountability for funding). The 
national evaluation occurs at the end of the fiscal year and reports results six months into 
the subsequent year. Formative evaluation is more immediately useful to guide 
programmatic decisions and therefore, more immediately relevant and useful. Going 
beyond the “bean counting,” which many respondents reported the indicators to be, may 
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mean reducing the focus of the questions or including better sampling strategies. They 
pointed to: 

• the dilemma of summative versus formative data, and making sure what gets done at 
the national level is explicitly useful at the local level 

• the stricter methodology at the national level, which may be at odds with the local 
level needs for formative information 

• a need for better balance between quantitative and qualitative methodologies 

• the fact that qualitative data are more valid for the local consortia - client voices 

Another area that would enhance the relevance for the local consortia is making sure 
we’re asking the right questions and collecting the most important data. Respondents 
emphasized the need to: 

• reflect on the evaluation work and revisit the assumptions about what needs to be 
done and reported both nationally and locally 

• refocus the priorities of OERI to figure out the meaningful questions politically and 
not just keep adding on 

• link the indicators to appropriate outcomes and determine what we really should be 
measuring and what we can realistically measure 

• reassess whether there is a need for as much detailed information as the national 
database requires 

• collect large-scale data every other year 

• align data collection and reporting cycles, as reporting times are not aligned between 
the national and local consortia 

• streamline the data collection process to collect only the data that are needed 

• explore conflict between the indicator-driven data collection and other data that may 
be more useful in strategic planning 

• receive more direction from OERI on what is important to them, to assist in 
evaluation planning 

• plan in advance what we want from the data, for example the v CCDDS, rather than 
asking many questions and deciding later which ones tell the best story 

• scale back or streamline indicators and focus on what is important; get directors and 
OERI to be clearer about which impacts and indicators are important 

One of the ways respondents saw to make the evaluation more relevant was to examine 
how we select samples for the evaluation. The biggest concern is that limiting what is 
collected from each consortium to feed into the national evaluation is really not as useful 
to the local consortia. Respondents called for: 

• increased sampling or better sampling 

• larger sample sizes, as small sample sizes don’t help the local consortia (doing one 

case study per consortium is not relevant) 

• thinking of sampling more from each project within consortia; narrowing the focus 
and rotating the focus over the course of the grant 
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The need for adaptability and flexibility in data ccllection tools . The consensus among 
the respondents is that the quality of the CCDDS database, surveys, and interviews is 
high. The only concern is the need for adaptability or flexibility to make the tools useful 
at the local level. One of the suggestions was to find out what else the local consortia 
would want to know on the surveys, interviews, and activity database. Another problem 
across the consortia is the compatibility of the database systems. Some consortia are PC- 
based and others are MAC-based. Respondents called for: 

• more flexibility in the kind of data that are collected 

• flexibility in how the national effort represents all the consortia, since it cannot 
represent any one consortium and when the local is evaluated on something it is not 
doing, it creates discomfort 

• tools to be more customizable to the individual consortium, such as PC-based versus 
Mac-based data systems 

• tying the national effort more closely to local work, as the specificity is lost from the 
local to the national level because of the need to be broad enough in what is collected 

• the local consortia to do the same data queries as the national, as it would motivate 
them to provide more accurate data, and they would see it as useful to them as an 
internal need rather than an external demand 

• the national evaluation to focus on output data and give the individual consortia as 
much flexibility with how the data are collected 

One way to enhance adaptability and flexibility is to seek more input and involvement 
from the staff around all parts of the evaluation process. Respondents suggested: 

• more input from staff and evaluation that is more closely aligned to what staff do 

• that results are put into a meaningful and marketable format; more communication 
with local staff before the national meeting; bringing staff questions and concerns to 
the national meeting 

• that program staff also communicate cross-consortia 

• more staff input and feedback to the tools and protocols that are used at the national 
level 

• that individual consortia add their own specific questions to the fall survey 

Quality of findings . Respondents talked about findings in terms of three key priority 
needs. One was the need to connect findings with research; the second was to discuss the 
application of findings; and the third was to do a better job of communicating the national 
findings to the local consortia. 

Some of the respondents noted that often the evaluation findings are simply reporting of 
results rather than an interpretation of the results as they relate to research. One 
respondent made the observation that the national evaluation work could also contribute 
to the research: 
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• more analyses and interpretation of results connected to research; more 
recommendations connected to research 

• developing a research perspective, which may differ from the national perspective, 
such as 150 hours of professional development rather than 12 or more hours 

• focusing evaluation on research base and tying results to the research 

• work that informs research, as what is learned from evaluation can fill research gaps 

A few respondents noted that often results were reported without any recommendations 
or suggestions for how to apply the results to the work of the local consortia. This is a 
result of aggregating the data and making sure that results are reported at the national 
level. Broad applications are global and less likely to be meaningful to the work of 
individual consortia. This requires that recommendations be made with the disaggregated 
data at the local level. Respondents called for: 

• focusing more on the application of the results on both the cross-consortia and local 
level 

• having a clearer set of recommendations and linking the recommendations to research 

Respondents suggested that they need to know the results of evaluation to even begin 
considering how to use the findings. Often reports are written in “evaluation-ese” and are 
not easily understood by non-evaluators. Respondents suggested: 

• making the results more relevant and easy to communicate to the local consortia; 
communicating the results in a meaningful and marketable way, so that staff are more 
likely “buy-in” to them 

• creating a packet of materials and sessions to disseminate to staff and for staff to 
disseminate to clients 

• informing the staff; “selling” them on why the national evaluation is important 

• helping staff understand the value of evaluation, so it doesn’t seem like a burden 

• that staff become more at ease with the data and able to connect the activities to 
clients in a better way 

• staff become more aware of evaluation 

Respondents generally believed that local evaluation work should be communicated and 
shared at the national level. There should be more articulation back and forth between the 
two levels of evaluation. Respondents pointed to: 

• sharing protocols at the national level and letting the local evaluation work guide the 
national rather than vice-versa as each level informs the other 

• sharing expertise back and forth more effectively as it has become too one-sided (the 
national has taken on more influence) 

• devising a way to compare the different models and strategies consortia are using to 
- -improve students’ mathematics and science performance . 

• the need for evaluators to share how they apply the results in their own consortium 
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C. Benefits of a National Evaluation Effort 

Although there are many areas to examine for improvement, many respondents offered 
unsolicited testimonials to the work of the evaluation committee. The consensus was that 
the national evaluation helped define a unified purpose to the work of the consortia by 
helping them focus their work to have a greater impact and that the quality of evaluation 
overall was improved by having a national peer group. Responses included: 

• I want to applaud the group. I think they 've been very serious about their work and I 
think I see it just getting better. 

• The measure of success is that the various directors have been willing to support the 
quality evaluation activities that the group has done. 

• It keeps the focus on the national priorities 

• It pushes us to focus on impact. 

• It has increased the rigor and quality of evaluation. 

• I like the idea of a national network. They try to keep us focused. Sometimes I think 
if it weren V for the network, each of us might be going off in to many different 
directions and not make an impact, so I do see the value. 

• I think the national evaluation has been a form of glue for the consortia because of 
the joint and shared goals. It 's helped us articulate our strands of work. It makes it 
look more like a national effort than ten individual efforts. 

D. Conclusions 

Each consortium approaches its work in unique ways that are designed to respond to the 
characteristics of their client base and the specific needs of the states, local districts, and 
individual clients in their regions. The challenge is to make the work at the national level 
of rigorous quality and relevant to the local consortia, so that scarce resources for 
evaluation are used effectively. The increased focus by local consortia on the impact of 
their work, particularly in the work with schools known as intensive sites, is a clear 
priority, as is the need to demonstrate the link between the work they do with teachers 
and education leaders and improved student performance. 

Interestingly, the focus of the suggestions for improving the quality and relevance of the 
evaluation at the national level did not address the need to examine impact. It talked 
more about making sure there was commitment and equity in the national committee to 
assure reliability and validity of the evaluation work. Most other recommendations 
focused on ways to get that increased commitment by making sure the national 
evaluation was not overly burdensome and more closely tied to what the consortia were 
doing in a formative way, which could be interpreted to mean help with assessing impact. 



As the national committee enters the ninth year of working together, the group needs to 
reflect on its own priorities and ways to keep the communication flowing, from the local 
to the national and back, to assure the best of both worlds. Many of the suggestions are 
imminently possible, while others are more complex and will require time for the 
evaluation committee to review and discuss them seriously. 
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